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ABSTRACT 


Tha  quantitative  prediction  and  measurement  cf  software  re¬ 
liability  is  cf  vital  Importance  in  the  development  of  high  quality  cost 
effective  software.  Many  software  reliability*  models  have  been  postulated 
in  the  literature  (Hef.  11),  however  few  have  been  applied  to  field  data# 

A  model  bated  upon  the  assumption  that  the  failur3  rata  cf  the  software  is 
proportional  to  ths  number  of  residual  software  errors  leads  to  a  constant 
failure  rate  and  an  exponential  reliability  function,  (Ref.  1).  Ths  model 
contains  two  constants i  the  proportionality  constant  IC  and  the  initial 
(total)  number  cf  errors  Ej. 

The  constants  K  and  can  be  estimated  during  early  design  'ey 
comparison  cf  the  present  project  with  historical  data.  During  the  in¬ 
tegration  test  phase,  a  more  accurate  determination  cf  the  model  parameters 
can  be  obtained  by  using  simulator  test  data  as  if  it  were  operational 


failure  data.  The  simulator  data  is  collected  at  two  different  points 
in  the  integration  test  phase  and  the  two  parameters  can  be  determined 
from  moment  estimator  formulas  (Ref.  9).  The  more  powerful  maximum 
likelihood  method  can  also  be  employed  to  obtain  point  and  interval 
estimates  (Ref.  3)«  lb  is  also  possible  to  use  least  squares  methods 
to  obtain  parameter  estimates  which  is  the  simplest  method  and  provides 
laslght  into  the  analysis  of  the  data  (Ref.  12). 


ill 


this  thesis  utilises  &  set  of  softvare  development  and  field 
data  taken  by  John  D.  Musa  (Ref .  10)  as  a  vehicle  to  study  the  ease  of 
calculation  and  the  correspondence  of  the  three  aethods  of  parameter 
estimation.  The  sensitivity  of  the  reliability  predictions  to  parameter 
changes  are  studied  and  compared  with  field  results. 

This  thesis  is  based  in  part  cn  a  joint  paper  written  by 
the  author  and  Professor  Martin  L.  Shopman ,  presented  at  the  CRSA-TIMS 
Conference  in  Florida,  January  1981.  (l4) 

The  results  show  that  if  data  Is  carefully  collected,  soft- 
ware  reliability  nodels  are  practical  and  yield  useful  results.  These 
can  serve  as  one  a  assure  to  help  in  choosing  aaong  competitive  designs 
and  aa  a  guage  of  when  to  terminate  the  integration  test  phase. 
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Introduction 


Software  presently  represents  the  highest  cost  item  in  the 
davelopcent  of  computer  systems.  There  is  &  paucity  of  quantitative 
measures  to  judge  the  quality  of  the  final  software  and  use  as  a  measure 
of  progress  during  the  test  and  debugging  phase.  The  reliability  and 
meantime  to  failure  (MTTF)  of  the  software  is  a  nost  useful  metric  for 
both  the  above  purposes. 

An  important  class  of  software  reliability  models  (see 
Refs.  1,8)  make  the  assumption  that  the  operational  software  failure 
rate  is  proportional  to  the  remaining  number  of  errors.  Thus  the  failure 
rate  is  dependent  on  development  time  T,  but  not  on  operating  time  t. 

This  leads  to  a  constant  hazard  and  exponential  reliability  model,  with 
two  unknown  parameters  K  and  E^. 

A  major  focus  of  this  thesis  is  to  investigate  and  provide 
insight  into  a  number  of  issues  related  to  the  estimation  of  these  two 
parameters  > 

1.  The  accuracy  one  can  obtain  by  using  historical  data  to 
'  determine  K  and  E^,  (Musa's); 

2.  A  comparison  of  the  accuracy  obtained  using  three  dif¬ 
ferent  methods  of  parameter  estimation:  the  maximum 
likelihood,  moments,  least  squares ; 

3.  A  comparison  of  predicted  values  of  MT  i'F  and  observed 
HTTP  values  from  field  failure  data; 

Model  parameter' sensitivity;  and, 

5.  Model  accuracy/parameters  related  to  the  practical  ap¬ 
plication  by  the  software  manager. 


-  -  n  -««.V  'jhWj »•  I  ■  lit  i  II 


The  conclusions  reached  in  this  study  dearly  indicate  that 
peraaeter  estimation  during  systaa  development  is  a  highly  practical  tool 
which  can  be  used  to  successfully  predict  software  MTT5*  and  the  'debugging' 
tine  required  to  achieve  that  goal* 
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2.0  Development  of  Error  and  Reliability  Models 

2.1  An  Error  Rea oval  Model 

The  reliability  model  used  in  this  paper  has  been  described 
in  detail  in  a  nun  her  of  references  (l,  2,  3)«  In  brief,  the  aodel  as- 
suaes  that  the  program  enters  the  integration  test  phase  with  Sj  total 
errors  resaining  in  the  software.  As  integration  testing  proceeds,  all 
detected  errors  are  promptly  corrected,  and  at  any  point  in  the  develop¬ 
ment  cycle  (after T months  of  development  time)*,  a  total  of  Ec  (t)  er¬ 
rors  have  been  corrected,  and  the  remaining  number  of  errors,  E;  is 

S(TJ  =  £t-£c(t)  (1) 

In  a  more  advanced  model  (4)  it  is  assumed  in  addition  that  new  errors 
are  generated  during  development.  Or. a  can  often  normalise  the  above 
equation  through  division  by  the  number  of  object  cede  instructions  Lj. 
to  yield 

Et( T)  -  _£r  _  Ed Tj  (2a) 

It  It  It 

€r(Tj=  £t-Cc(t}  (2b) 

It 

'Where  €f=JLi  and  €c 

h  It 

Basically  the  error  model  used  in  this  paper  assumes  that  the 
total  number  of  errors  in  the  program  is  fixed  and  that  if  we  record  the 

*  In  some  cases  the  actual  number  of  test  hours  is  estimated  and  is  used 
as  the  development  time  variable  rather  than  the  cruder  calendar  days. 


cumulative  number  of  errors  corrected,  during  debugging,  then  the  difference 
‘  represents  the  remaining  errors.  We  can  define  error  removal  rates  ast 


ag,  T 
d  T 


vhich  can  be  normalized  to  yield 


(3) 


PrW 


where 

P,(T)  “  errors  removed/total  nuaber  of  instructions/test  hours  (5) 

MX)  -  J* p (x)dx  -  cumulative  errors/ total  nuaber  of  Instructions,  (6) 
In  Reference  5  error  data  are  reported  for  seven  large  super¬ 
visory  programs  and  applications  programs.  In  Fig,  1  the  normalized  error 
rateP(T)  calculated  from  this  data  is  plotted  as  a  function  of  T,  the 

f 

number  of  months  of  debugging  after  release  for  three  of  the  seven  systems. 
Although  several  curve  shapes  might  be  fitted  to  this  data ,  one  character¬ 
istic  is  common  for  all  curves.  The  normalized  error  rate  decreases  over 
the  entire  curve  or  at  least  ever  the  latter  two- thirds  or  half  of  the 
curvet  whereas  initial  behavior  c £  P(T)  differs  from  example  to  example. 

A  curve  of  the  cumulative  error  data  for  the  supervisory  system  A  of  Fig.  1 
is  shown  in  Fig,  2.  Similar  curves  of  €  (Z)  drawn  for  the  other  examples  of 
Fig.  1  all  build  up  Initially  with  a  constant  or  increasing  slope  and  then 
exhibit  a  decreasing  curvature  appearing  to  become  asyr.pt otic.  The  smo¬ 
othing  nature  of  integration  makes  all  the  €  (T)  curves  lock  more  alike  than 
theP(T)  curves  do.  (Figs.  2A  A  B).  3oth  are  needed  for  a  detailed  study. 


4  5  6  7  012345678  012345 


O  I  2 
r  months 


Supervisory  A  Supervisory  8  Supervisory  C 

FIGURE  I.  NORMALIZED  ERROR  RATE  vs  DEBUGGING  TIME 
”  FOR  THREE  SUPERVISORY  PROGRAMS 


Mugging  Time  -  r-  Months 

FIGURE  2.  Cumulative  Error  Curve  for 
Supervisory  System  A  Given  in  Figure  I. 
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If  we  assume  that  the  total  number' of  errors  in  the  program 
(E,j)  Is  constant  and  that  the  program  contains  Ij  instructions,  then  the 
asymptote  which  the€(T)  curves  approach  is  E^/lj.  (Figs.  2,  2A,  23). 

The  error  model  developed  above  will  be  used  in  the  following 
section  to  formulate  a  reliability  model. 

2.2  Development  of  the  Exponential  Reliability  Model 

We  assume  that  all  operational  software  errors  occur  due  to 
the  occasional  traversing  of  a  portion  of  the  program  in  which  a  hidden 
software  error  is  lurking.  He  begin  by  writing  an  expression  for  the 
probability  that  an  error  is  encountered  in  the  time  interval  At  after  t 
successful  hours  of  operation.  We  make  the  assumption  that  this  prob¬ 
ability  is  proportional  to  the  number  of  errors  remaining  in  the  program. 
(See  Ref.  6  for  data  substantiating  this  assumption). 

From  a  study  of  basic  probability  and  reliability  theory  we 
know  that  the  probability  of  failure  in  time  interval  t  to  t  +  At,  given 
that  no  failures  have  occurred  up  till  time  At,  is  proportional  to  the 
failure  rats  (hazard  function)  z(t).  Thus,  we  obtain 

P(t<tf<t  +dt  |  tf>t)  -  z(t)dt  -  K€r(T)dt  (7) 

where  t^  -  time  to  failure,  (occurrence  of  a  software  error) 

F(t<tf<t  +dt|t£>t)*  probability  of  failuro  in  interval  At, 

given  no  previous  failure. 


K  -  an  arbitrary  constant 


Frco  reliability  theory  (?)  we  can  show  that  the  probability 
of  no  system  failures  in  the  interval  0  to  t  is  the  reliability  function 
which  is  related  to  the  hazard,  z(t),  byi 

/?W  =  «-£2W*  (8) 

If  we  substitute  our  expression  for  2(t)  frcx  Eq.  ?  into 
a**  8  and  assume  K,  and  6^  (t),  are  independent  of  prorating  tine,  we 
obtain 

=  .‘r'  W 

Basically  the  above  equation  states  that,  the  probability  cf 
successful  operation  without  software  errors  is  an  e.vr.onentlal  function 
of  operating  time.  When  the  system  is  first  turned  on,  t  -  0  and  H(o)  •  1, 
As  operating  time  increases  the  reliability  monotonicv.lly  decreases  as 
shown  in  Fig.  3«  We  depict  the  reliability  function  for  three  values  cf 
debugging  time,  .  From  this  curve  we  nay  irc.I:e  various  pre¬ 

dictions  about  the  system  reliability.  For  example,  looking  along  the 
vertical  line  t  -  \/y  we  cay  3tato» 

1.  If  we  spend  Tq  hours  of  debugging  then  R(l/yO  ■  0.35 

2«  If  we  spend  houru  of  debugging  then  H(l/y)  *0.50 

3«  If  we  spend  T.,  hours  of  debugging  then  R(l/^)  "0.75 
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Tj>>  ij  most  debugging 

T^t^most  debugging 
Tq  leost  debugging 


FIG.3 


NORMALIZED  OPERATING  TIME  t 

Variation  of  Reliability  Function  R(t )  with 
Oebugging  Time  r. 


a.  Constant  Error  Debug  r.ste 
FIGURE  4 ,  Comparison  of  MTTF  vs 
Debug  Time  for  The  Error  Model. 


10 


2*3  Meantime  to  Software  Failure 

A  simpler  way  to  summarize  the  results  of  the  reliability 
aodel  is  to  compute  the  mean  tine  to  (software)  failure ,MTTF,  for 
the  system* 


(10) 


For  purposes  of  illustration  we  latp  (t)  be  modeled  by 
a  constant  rats  of  error  correction  PQ,  Solution  of  Eqs.  2  and  10 
then  yields 


where  (3  -  K  Bp/lT  and  Ct~  P0VSr 

This  is  depicted  in  Fig.  4  where^xMTTF  is  plotted  vsqxv 
and  we  note  that  the  greatest  improvement  in  HTTP  occurs  during  the 
last  i  of  the  debugging  stage. 

Reference  5  describes  other  error  correction  rate  models , 
as  illustrated  in  Fig.  ^A.  In  order  to  compare  the  relative  effects  on 
system  MTTF  by  assuming  a  varying  p(r),  it  is  essential  that  we  integ¬ 
rate  the  latter  over  the  same  range.  This  is  readily  verified  by 
noting  that  the  total  area  under  each  curve  is  identical.  Integration 
results  are  plotted  in  Fig.  4B  for  each  model  to  yield  the  cumulative 
number  of  expected  errors.  The  effect  on  system. MTTF  is  depicted  in 
Fig.  4C. 


S"ignro  5. 


Growth  Curr«  o £  So£iware  ^eliabiiity  Mean 
Time  Between  Se£r»nr*  Error 3.  (From  X. 
Miyamoto,  Ref.  U). 


MTBF  — HOURS 


The  rapid  rise  In  HTTP  toward  the  «nd  of  tho  integration 
phase  is  of  considerable  importance  in  software  project  management. 

Held  data  by  others  (see  Pigs.  ^  &  6)  confirms  this  basic  shape.  If 
a  manager  is  pressed  to  release  a  system  to  the  field  at  an  early  date, 
he  nay  accept  the  current  reliability  and  deliver  the  software.  If  we 
believe  Figs.  4  thru  6,  however,  then  a  few  sore  weeks  could  yield  a 
big  improvement  in  HTTP.  The  model  predicts  such  behaviour,  but  if 
one  only  had  test  data  for  QT£t#  then  it  would  be  difficult  to  pre¬ 
dict  the  sharp  rise  near QT-1 .  The  model  is  thus  of  great  use  in 
managing  a  project  and  setting  its  release  date. 

2.4  Musa's  Model 

Musa  has  developed  a  model  similar  to  that  £iven  in  Sec.  2.2. 
However,  instead  cf  basing  his  model  on  development  tiro  as  the  resource 
measure  during  integration,  he  utilized  actual  CRJ  time.  Musa  also  used 
regular  test  data  rather  than  simulator  test  data.  His  model  is  given 
by  (8)  i 

BCV)  -  e“T/T  (12) 


7^e-(cVMoTo) 


(13) 


^5 


1 


where 

T*  •  hours  of  program  operation  -  t 
T  -  mean  time  to  failure  ir.  operating  hours  “  HTTP 
T0  -  HTTP  at  the  start  of  test  (t  ■  0) 

C  -  ratio  of  equivalent  operating  time/test  time 
M0  -  number  of  failures  which  must  occur 
to  uncover  all  errors  *  Em 
X  -  the  CRJ  time  in  hours  during  testing 

Since  ve  will  be  using  Musa's  data  and  same  of  his  results 
In  Sec.  5  of  "this  thesis,  we  must  carefully  account  for, the  different 
.definitions  of  time  between  Musa's  model  and  the  exponential  model 
during  the  analysis  of  his  data. 


Estimation  of  Model  Panr.atars 
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3«1  Introduction 

The  exponential  reliability  model  given  in  Eq.  9  contains  the 
parameters  K  and  €r(t)  which  must  be  estimated.  In  many  cases  one  wishes 
to  use  such  a  model  to  roughly  predict  MTTF  during  a  proposal  phase  or 
early  design  of  the  project.  In  such  a  ease  the  only  available  technique 
for  determining  values  of  the  parameters  is  to  use  historical  data. 
Presently  Rome  Air  Development  Center  is  developing  a  handbook  and  database 
on  Software  Reliability  for  just  such  a  purpose. 


3.2  Moment  Estimates 

In  Ref.  9,  a  method  is  discussed  for  measurement  of  the  two 
needed  parameters  based  on  simulation  testing  of  the  software.  A  program 
simulating  the  field  environment  is  generally  available  for  all  real  live 
computer  programs.  It  is  necessary  that  this  program  be  run  for  a  total 
of  hours  following  months  of  development.  It  is  assumed  also  that 
the  testing  will  produce  r^  failures.  Similarly  after  T2  months  of  testing, 
the  simulator  is  run  and  Hg  hours  and  r^  failures  arc  obtained.  The  MTTF 
for  the  data  is  given  by  the  ratio  H^/r^  and  is  equated  to  the  MTTF  ex¬ 
pression  obtained  by  substituting  Eq.  9  into  Eq.  10.  The  two  equations 
(for  and  T^)  allow  us  to  solve  for  the  constants  K_and  E^t 


K[Sx  -  EcCT^J 
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Simultaneous  solution  of  Eqs .  Ik  and  1$  yields  the  desired  values  of 
£p  and  K: 


(Vx,-') 


(16) 


_ 


(17) 


Note  that  in  deriving  the  above  results  we  have  assured  that  the  failure 
late  is  constant  and  have  used  the  c can on  notation  and  well  known 
results  for  constant  failure  rates i 


*(t)  -  x  -  -5- 


3*3  Least  Square  Estimates 

r 

Another  method  of  estimating  model  para&c;:  rrs  is  to  rewrite 
equation  11  in  the  form 

X;  -  k\et  -  Ee  faj|  (18) 

for  X"  failure  rate 

to  yield  Ec  (TA)  -  E^-  4-^  (19) 

This  equation  represents  a  straight  line  ?:.':oso  parameters 
can  be  determined  from  the  slope  and  intercept  of  a  Ic^-nt  squares  fit  of 
the  data,  where 
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-  intercept 


Naturally*  the  largerand  the  aore  accurate  the  data,  set*  the 
■are  precisely  can  the  model  parameters  be  predicted* 


3*4  Maximum  Likelihood  Estimates 

Another  method  which  can  also  be  used  to  estimate  the  values 

of  X  and  is  known  as  the  Maximum  Likelihood  Estimation  technique  (KLE) . 

The  likelihood  function,  L  is  the  joint  probability  of  occurrence  for  the 

observed  set  of  test  values.  If  during  simulation  testing  we  observe  r^ 

failure  times  (t^.tj,  ...tr  )  and  n^-r^  successful  runs  testing  (T., , 

•  ••»  T„  _-r  )  then  the  likelihood  function  for  a  single  test  after  E  (r, ) 

c  i 

errors  have  been  removed  is  given  by 


where 


LOC,!^)  -  f(t1)f(t2)....f(tri)nCt1)R(T2)....H(T^r^ 
f(t^)  •  the  density  function  KEr ( )  e~^f» w*  J  \ 

HC^)  -  the  reliability  function  e"K€f(~’jTi 


To  maximize  the  likelihood  function  we  take  partial  devivitives  with 
respect  to  X  and  E^  and  set  them  equal  to  zero*  To  solve  for  the  two 
operations  we  need  a  second  equation  obtained  from  another  likelihood 
equation  based  on  a  second  set  of  test  data  at. time  T,., .  Applying  KLE 


theory  to  two  testa  with  and  r2  failures  over  and  total  hours, 
we  obtain  (Appendix  A,  Ref.  3) 

K  - 

* 

K  - 

As  Is  often  the  case  with  MIS,  the  above  equations  require 

numerical  solution i  however,  most  statisticians  believe  then  to  yield 

superior  results  to  aonent  estimates .  An  iterative  computer  solution 

of  Eqs.  22  and  23  is  easily  implemented,  y3t  a  graphical  solution 

using  a  staple  calculator  suffices  in  most  cases*  The  first  step  is 

»  * 

to  obtain  starting  values  for  Sj  and  K  using  some  other  method,  such 

«• 

as  Method  of  Moments .  Values  cf  2^  above  and  below  the  starting 
value  are  substituted  into  Eqs.  22  ,  23  and  the  curves  of  K  vs  E^, 
plotted  on  the  sane  axes.  Their  intersection  determines  the  value 


Musa's  Data 
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4.0 


4.1  Introduction 

The  software  reliability  data  used  in  this  paper  was  compiled 
by  John  D.  Musa  (lo)  over  a  period  of  time  on  a  variety  of  large  software 
systems,  ranging  from  tens  to  hundreds  of  thousands  of  object  code  in¬ 
structions.  Fis  objective  was  "to  present  in  detail  a  substantial  body 
of  data  that  has  been  gathered  in  the  application  cf  the  execution  tire 
theory  of  software  reliability" }  the  end  product  is  ideally  suited  for 
the  purpose  of  this  paper,  as  it  presents  a  wealth  of  precise  software 
failure  data  obtained  under  carefully  controlled  circumstances. 


4.2  Description  of  Haw  Data 

Software  failure  interval  data  was  presented  in  tne  following 


format < 

Failure  number  Failure  Interval  Day  of  failure 

Failure  interval  was  measured  in  seconds,  and  represents  either  running 
clock  time,  (operating  tise  on  the  computer),  or,  in  one  case,  actual 
CHJ  time,  'Day  of  Failure'  is  the  working  day  counted  from  the  start 
of  project  on  which  the  failure  occurred.  (See  Table  l). 


TABLE  1 

ILUFE  INTERVALS  -  SYSTEM  3  TEST  PHASE 


Failure 

Number 

Failure 

Interval 

Day  of 
Failure 

Failure 

Number 

Failure 

In  ter/a  1 

1 

115 

1 

21 

15 

2 

0 

1 

22 

390 

3 

83 

3 

23 

1863 

4 

178 

3 

24 

1337 

5 

194 

3 

25 

4508 

6 

136 

3 

26 

834 

7 

1077 

3 

27 

3400 

8 

15 

3 

28 

6 

9 

15 

3 

29 

4561 

10 

92 

3 

30 

3186 

11 

50 

3 

31 

10571 

12 

71 

3 

32 

563 

13 

606 

6 

33 

2770 

14 

1189 

8 

34 

652 

15 

40 

8 

35 

5593 

16 

788 

18 

36 

11696 

17 

222 

18 

37 

6724 

18 

72 

18 

38 

25^6 

19 

«i5 

18 

39 

10175* 

20 

589 

26 

*  flote»  If  the  last  Interval  Is  followed  by  an  asterisk,  there  was 
no  failure  at  the  end  of  the  period  and  the  tine  represents  the 
interval  between  the  last  failure  and  the  end  of  the  period,  (lo) 
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Par  the  purpose  of  this  paper ,  failures  occurring  on  the  sane 
working  da/  were  sunned  together  to  fora  one  statistical  data  point,  and 
were  tabulated  under  the  following  format i  (See  Table  2) 


t  5  '  T~  liiT  1 


T"T 


a  c 


where t 

S  »  sequential  serial  number  assigned  to  each  statistical 
data  point 

VO  •  working  day  on  which  faJLlure(s)  occurred 
E  «  number  of  errors  occurring  that  working  day 
Ec  •  cumulative  errors  to  date 

T  »  total  operating  time  (failure  interval  tine)  for  that 
working  day 

To  •  cumulative  failure  interval  time  to  date 
s  -  failure  rate  (2/T)  for  that  working  day 


Presentation  of  Musa's  data  in  this  format  had  a  twc-fold 

purpose i 

a)  reduce  the  sheer  bulk  of  the  raw  data  without  affecting 
its  statistical  significance  by  grouping  the  occurrence 
of  software  failures  by  working  day;  and, 

b)  tabulate  the  data  in  a  format  more  suitable  for  subsequent 
calculations. 


4,3  System  Characteristics 

Systems  1*4  studied  by  Musa  are  real-time  command  and  control 
software  packages  consisting  of  21,700  to  33*500  object  instructions  and 
a  failure  sample  size  of  38  to  1 36. 

System  5  is  &  realtime  commercial  application,  consisting  of 
2,445,000  object  instructions  and  a  failure  sample  size  of  831*  Musa 
notes  that  for  system  5*  design  changes  involving  2l£  of  the  source 
code  were  introduced  after  approximately  30?5  of  total  testing  time  had 
elapsed. 
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5*0  Estimation  of  Modal  Constants 

In  the  following  section*  we  will  be  estimating  the  nodal 

A  4* 

constants  K  and  of  several  system,  using  system  if 3  as  a  working 
example  to  demonstrate  the  calculations  used  for  Least  Squares*  Method 
of  Moments ,  and  Maxinun  Likelihood  Estimates.  These  constants  will 
then  be  used  to  estimate  system  MTTF  using  equation  11. 

As  indicated  in  Sec.  4.2,  Musa's  ra>  data  was  condensed  to 
form  one  statistical  data  point  per  working  day  during  which  one  or 
more  errors  were  uncovered. 

In  the  following  section*  this  condensed  data  was  reduced 
even  further  to  allow  2,  3.  4,  6,  8...  points  to  represent  an  entire 
system  as  required.  For  example,  if  it  were  desired  to  represent 
system  3  (see  Table  2)  consisting  of  18  entries,  by  two  points  (Fig.  7), 
the  first  nine  entries  were  averaged  statistically  to  yield  the  first 
point  and  the  last  9  entries  to  yield  the  second  point. 

5*1  Method  of  Moments 

The  data  reported  by  Musa  for  system  #3  has  been  processed 
and  grouped  by  working  day  as  described  in  Sec.  4.2  (see  Table  2).  Using 
data  points  fcr  the  9th  and  18th  group  of  failures,  (Ec  9,  Tc  9), 

(Ec  18,  Tc  18),  we  obtain  the  average  failure  rate  fcr  the  interval  0-9* 

Xl  -  "  0.C0175  failures/sec.  -  6.3  failures/hr. 

sec. 

and  similarly  the  average  failure  rate  for  the  interval  10-18  is  i 

X ,  -  gSl§L  Z.Efi9  . 

2  Tcl8  -  Tc9  67,362  -  14,260 

-  2.448  x  10“**  failures/sec.  -  0.881  failures /hr. 


System 

1 

Ec  -  136 
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Bc9*25 


Ecie  ‘  39 


T=9  -  14,260 


Tel8  -  67 • 362 


v.  plot 


x,  -  - 17.53  x  io‘ 

*  14,260 


*1  -  Sc9  *  25 


I,  -  2.  “  2-45  *  10"4  12  -  Scl8  -  38 


2  2  67, 362-14, Z60 


to  yield 


Ej  -  40.1 


K  -  1.1603  x  10* 


The  above  calculations  are  depicted  in  Fig.  7  far  system  3 
and  summarized  for  all  systems  In  Table  4. 


5.3 


Maximum  Likelihood  Estimates  (MLS) 

The  primary  equations  used  (see  Sec.  3.4)  arsi 


K.  - 


ri  +  r2 


1  «lCVEclJ+H2(VE=2> 


(22) 


*2  xp hj 


1  ♦  *2 

EZ^B  w^5r 


v&ci  w 


(23) 


CUMULATIVE  ERRORS  E 
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TABLE  4 

LEASE  SQUARES  REGRESSION  ESTIMATES  0?  Ey,  K 
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A  numerical  solution  of  the  above  equations  can  be  readily 

A  * 

obtained,  since  we  have  two  equations  in  two  unknown,  K  and  E*.  A 
simpler  approach  assuming  we  only  have  a  few  estimates  to  Bake,  is  to 
effect  a  graphical  solution! 

Step  it  obtain  a  starting  value  for  Ey  by  any  method  of 
your  choice!  Method  of  Moments ,  Least  Squares 
Regression,  Son-deterministic  (ie...  guess), 

Step  2 i  substitute  a  value  Ey  obtained  in  Step  1  into 
equations  22  and  23 •  Try  other  values  of  Ey 
above  and  below  this  value  and  repeat  the  cal¬ 
culations,  plotting  them  on  the  sane  axis  of 
* 

K  vs  Ey  for  the  two  equations;  and, 

Step  3*  the  intersection  of  these  two  curves  yields  the 
required  parameters. 

The  method  is  illustrated  using  system  3  as  an  example! 

Step  li  Ey  -  40,1  using  the  starting  value  obtained  by 
the  Method  of  Moments  (MOM)  Table  3. 


Step  2i  For  Eyy  -  39,  K  ■  1.504  x  10“^ 

(from  22) 

Kg  -  2.195  x 

(from  23) 

For  Eyg  -  41,  Ky  -  8.355  *  10’5 

(from  22) 

K.  -  7.008  x  10‘5 

(from  23) 

Step  3»  The  above  results  are  plotted  on  Fig.  8  to  yield 
Ey  -  40.1 
£  -  11.6  x  10‘5 


Once  an  approximate  value  is  obtained  frc:;  the  curves'  in¬ 
tersection,  iterative  methods  can  be  used  to  obtain  the  required  degree 


ESTIMATED  KxlO 
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1 


MAXIMUM  LIKELIHOOD  ESTIMATES  CP 


System 


A 

A 

Em 

X 

T 

151.59 

5.141  x  10*5 

61.61 

3.909  x  10"5 

40.11 

I.I60  x  10"4 

I 

54.80 

,  ! 

2.249  x  10^ 

6,0  Model  Jkraaeter  Accuracy  and  Sensitivity 

6.1  Introduction 

Several  additional  factors  must  be  considered  when  using 
this  exponential  model  -  or  any  other  aodel  for  that  matter  -  in  pre¬ 
dicting  software  system  performance. 

Two  aspects  considered  in  this  thesis  aret  ' 

1,  Model  Accuracn  The  more  data  one  has  on  a  particular 
system,  ths  more  confident  one  can  be  that  the  parti¬ 
cular  aodel  predictions  will  approximate  reality j 

2,  Model  Sensitivity!  When  software  field  failure  data 

is  available  to  compare  with  model  prediction  estimates, 
it  seems  -  initially  that  slight  changes  in  model  para¬ 
meters  are  not  proportionally  reflected  in  nedel  pre¬ 
dictions  for  system  HTTFj 

6.2  Model  Accuracy 

Intuitively,  one  feels  that  the  more  data  one  has  about  a 
particular  system,  the  more  accurate  will  be  our  predictions  about 
system  performance.  To  a  certain  extent  this  is  true.  This  implies, 
however,  that  model  prediction  accuracy  only  approaches  reality  as  we 
near  the  final  debugging  stage,  whereas  the  'software  manager'  is  quite 
interested  in  estimating  software  reliability  during  early  testing 
stages  to  determine  future  trends.  As  noted  in  Section  2*3,  this  is 
strongly  implied  by  Fig*  4  which  indicates  that  the  maximum  return  for 


one's  efforts  is  obtained  only  during  the  latter  2$%  of  the  debugging 
phase. 

Hindsight  will  usually  enable  us  to  look  back  at  a  parti¬ 
cular  software  development  project  and  to  announce  that  'x'  time  rather 
than  'y'  time  should  have  been  spent  on  development  and  testing. 

The  onset  of  the  release  point  can  be  determined  in  several 
ways,  two  of  which  are  toi 

a)  closely  nonitor  the  slop  of  our  MTTF  curve  plotted 
versus  time  spent  on  debugging  and  to  watch  for  a 
sharp  increase  as  it  nears  a  vertical  asymptote 
(Fig.  4,  Uc)  or, 

b)  since  the  system  MTTF  is  directly  related  to  the 
number  of  software  errors  remaining  (S^  -  Ec),  Icok 
for  the  point  in  time  when  model  predictions  of 
applied  to  the  system  in  question  approach  a  hori¬ 
zontal  asymptote.  This  aspect  13  depicted  in  Fig.  9 
by  the  shape  of  expected  value  and  ^Tccnfidence  bands 
around  as  testing  nears  completion. 

6.3  Model  Sensitivity 

The  un-normalized  equation  used  to  calculate  system  MTTF  is 
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snd  the  derivative  with  respect  to  K  Is 


a  HTTP  -  -1  ,  _  HTTP 

“TT  K2(3j  -  ec(-))  K 

This  indicates  that  variations  in  K  result  in  a  quasi-linear  effect  on 


(28) 


HTTP.  For  snail  variations  in  K  this  variation  can  be  linearized  as 


shown  in  Fig#  10  where  the  percentage  change  in  K  is  plotted  vs  resulting 


percentage  change  in  HTTP# 

m 

alight  variations  of  E^#  however#  result  in  dramatic  changes 
in  HTTP,  especially  as  the  former  approaches  the  value  of  E^Cr).  Cal¬ 
culating  c)  MTTF  yields  t 
k)  Et 


c)  MTTF  -1  .  _  HTTP 

k(Bt  -  ec(z)Y  (fip  -  Ecirj) 


(29) 


Thus#  as  2c(rJ->£pf  the  sensitivity  becones  quite  large.  It 
can  be  seen  from  Fig.  11  that  a  10jS  change  in  3 T  results  in  70%+  variation 
of  system  MTTF.  This  factor  alone  emphasises  the  requirement  of  high- 
quality  failure  data  if  accurate  model  predictions  are  to  ensue. 


TESTING  TIME  (hrs.) 
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7*0  Conclusion 

Tha  primary  air.  of  this  thesis  was  to  present  a  model  of 
software  systea  parameter  estimation,  to  subsequently  compare  predicted 
system  characteristics  with  actual  field  data,  and  to  guide  the  software 
manager  in  the  practical  application  thereof  to  software  systems  under 
his  development. 

Table  6  summarizes  model  parameters  by  system  and  method  of 
calculation.  It  can  be  seen  that  KTTF  prediction  accuracy  varied  from 
2JS  to  6?%,  depending  on  the  system,  with  an  overall  average  of  45JS. 

Surprisingly  enough,  there  was  less  than  difference  in 
prediction  accuracy  between  the  three  different  methods  of  model  para¬ 
meter  calculation  Method  of  Moments,  Least  Squares,  and  Maximum  Likeli¬ 
hood  Estimates.  Although  most  statisticians  consider  MLE  to  yield 
superior  results  to  other  methods,  the  findings  of  this  thesis  would 
indicate  that  parameter  estimation  using  Least  Squares  is  preferable 
for  the  software  manager  due  to  the  less  cumbersome  calculations  re¬ 
quired. 

Model  parameter  sensitivity  was  explored  and  indicated  that 
* 

&  variation  in  X  resulted  in  a  linear  variation  in  estimated  MTTF.  This 
was  not  the  case  with  variations  in  total  systea  estimated  errors  (Ep), 
however,  as  minute  variations  in  the  latter  resulted  in  drastic  changes 
in  predicted  KTTF. 

Once  model  parameters  and  systea  MTTF  have  been  calculated, 
it  is  of  great  interest  to  the  software  manager  to  uso  these  calculations 
to  determine/ predict  the  optimal  release  date  for  the  system  under  develop¬ 
ment.  Sections  2*3  and  6,2  suggest  several  ways  of  doing  so. 
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cona&isoN  of  model  HtDicriu.5  .xth  ftsld  assassa 


j 

!  Syat.  No. 

1 

1 

m 

B 

T 

z 

e 

kttft 

■ 

(CxHTI^y) 

KTTS*f 

<4,  i  HTT?- 

n  Kkusa'3’ 

1 

2  ! 

i 

1 

1 

MCM 

5*514 

152 

136 

.323 

15.1 

4.9 

14.6 

1 

66.6  ]  20.4 

1 

39.7  j 

IS 

5.515 

151 

•  323 

4.9 

66.6 

! 

MLS 

5.141 

152 

.346 

5‘2  L 

64e2  1 

| 

! 

!  2 

3.909 

62 

54 

.935 

13.6 

12.7 

31.4 

59.5  j  43.5 

38.5  1 

3.909 

62 

.935 

12.7 

59-5 

i 

3.909 

62 

.934 

12.7 

59.6  | 

; 

!  3 

11.60 

40 

38 

1.14 

13.2 

15.0 

30.3 

50.4  J  30.4 

0.33 

11.603 

40 

1.14 

15.0 

'50.3  : 

11.60 

40 

1.13 

15.0 

50.6  ! 

1  ' 

! 

4 

19.88 

55 

53 

.735 

13.1 

9.63 

9.17 

5.1  14.5 
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